Papers with loss functions
Contrastive Data and Learning for Natural Language Processing (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on effective representation learning algorithms. |
| Approach: | This tutorial introduces contrastive learning and provides an introduction to the techniques. |
| Outcome: | This tutorial provides an introduction to the fundamentals of contrastive learning approaches and the theory behind them. |
Multi-Surrogate-Objective Optimization for Neural Topic Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Neural topic modeling incorporates multiple loss functions but can be difficult to optimize for disparate magnitudes of these losses. |
| Approach: | They propose a gradient-based multi-objective optimization approach that integrates MOO algorithms into the model without the need for hard-parameter sharing. |
| Outcome: | The proposed approach outperforms direct MOO applications on NTMs. |
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent studies suggest multimodal transformer models learn rich visual-linguistic representations. |
| Approach: | They focus on dataset noise and language similarity to their downstream task . they find that models with a multimodal attention mechanism outperform deeper models with modality-specific attention mechanisms. |
| Outcome: | The proposed models outperform models with a multimodal attention mechanism on downstream tasks. |
Rethinking Loss Functions for Fact Verification (2024.eacl-short)
Copied to clipboard
| Challenge: | Existing objective functions for fact verification fail to capture heterogeneity among verdict classes . cross-entropy loss treats all misclassification types uniformly, which is problematic . |
| Approach: | They propose two task-specific objective functions that capture the heterogeneity among verdict classes . they use a dictionary-based objective function to classify Wikipedia sentences into three verdict classes. |
| Outcome: | The proposed objectives outperform the standard cross-entropy loss objective . the proposed objectives are combined with simple class weighting to overcome imbalance . |
Extractive Entity-Centric Summarization as Sentence Selection using Bi-Encoders (2022.aacl-short)
Copied to clipboard
| Challenge: | Entity-centric summarization is a type of controllable summarizing that aims to produce a summary specific to a given target entity. |
| Approach: | They propose to recast a sentence selection task as a controllable summarization using a dataset supported by EntSUM. |
| Outcome: | The proposed framework outperforms the current state-of-the-art in the sentence selection task and outperformed the competitive entity-centric Lead 3 heuristic by 1.1 F1. |
What to Learn, and How: Toward Effective Learning from Rationales (2022.findings-acl)
Copied to clipboard
| Challenge: | Increasing interest in learning from rationales has led to the use of human-annotated explanations to inject useful inductive biases into models. |
| Approach: | They propose several novel loss functions and learning strategies to exploit human rationales to augment model prediction accuracy. |
| Outcome: | The proposed learning strategies improve on three datasets with human rationales and show that they are more efficient than baselines. |
Can Brain Signals Reveal Inner Alignment with Human Languages? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Brain Signals, such as Electroencephalography, and human languages have been explored independently for many downstream tasks, however, the connection between them has not been well explored. |
| Approach: | They introduce a multimodal transformer alignment model to observe coordinated representations between EEG and language. |
| Outcome: | The proposed method achieved an F1-score improvement of 1.7% on ZuCo and 9.3% on Zuco datasets for sentiment analysis, and 7.4% on ZuCO for relation detection. |
Enhancing Discriminative Representation in Similar Relation Clusters for Few-Shot Continual Relation Extraction (2025.naacl-long)
Copied to clipboard
Anh Duc Le, Nam Le Hai, Thanh Xuan Nguyen, Linh Ngo Van, Nguyen Thi Ngoc Diep, Sang Dinh, Thien Huu Nguyen
| Challenge: | Existing methods for relation extraction (RE) fail to address the problem of similar relations, which contributes to catastrophic forgetting. |
| Approach: | They propose a relation extraction method that utilizes relation descriptions and dynamic clustering to identify similar relations. |
| Outcome: | The proposed method mitigates catastrophic forgetting and outperforms state-of-the-art methods by a large margin. |
A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | Non-autoregressive translation models suffer from the multi-modality problem when a source sentence corresponds to multiple correct translations. |
| Approach: | They propose to decompose the syntactic multi-modality problem into short- and long-range models and evaluate them on synthesized and real datasets. |
| Outcome: | The proposed loss functions can handle short- and long-range syntactic multi-modalities better than existing models. |
Sentence-Level Resampling for Named Entity Recognition (2022.naacl-main)
Copied to clipboard
| Challenge: | named entity recognition (NER) tasks are often dominated by the majority of non-entity tokens in text . a data imbalance problem is causing the NER models to ignore named entities . |
| Approach: | They propose a set of sentence-level resampling methods to reduce data imbalance . they use a training sentence to compute the importance of each training sentence based on its tokens and entities . |
| Outcome: | The proposed methods outperform sub-sentence-level resampling, data augmentation, and loss functions on multiple corpora. |
An Evaluation of Disentangled Representation Learning for Texts (2021.findings-acl)
Copied to clipboard
| Challenge: | Disentangled representations of texts encode information pertaining to different aspects of the text in separate vector embeddings. |
| Approach: | They propose to use a highly-structured natural language dataset to evaluate disentangled representations for texts. |
| Outcome: | The proposed models are well-suited for learning disentangled representations of texts on a synthetic natural language dataset. |
A Multi-answer Multi-task Framework for Real-world Machine Reading Comprehension (D18-1)
Copied to clipboard
| Challenge: | Existing models of machine reading comprehension (MRC) are based on cloze style questions or crowdworkers given a short passage from well-edited sources. |
| Approach: | They propose a multi-answer multi-task framework that uses multiple reference answers for multiple questions. |
| Outcome: | The proposed model increases the ROUGE-L score on the DuReader dataset from 44.18, the previous state-of-the-art, to 51.09 . |
Exploring the Trade-off Between Model Performance and Explanation Plausibility of Text Classifiers Using Human Rationales (2024.findings-naacl)
Copied to clipboard
| Challenge: | Saliency post-hoc explainability methods are important tools for understanding complex NLP models, but they may not align with human intuition, making the explanations not plausible. |
| Approach: | They propose a method for incorporating rationales into text classification models by augmenting the standard cross-entropy loss with a novel loss function inspired by contrastive learning. |
| Outcome: | The proposed approach enhances the plausibility of post-hoc explanations while preserving their faithfulness. |
Exploring Ordinality in Text Classification: A Comparative Study of Explicit and Implicit Techniques (2024.findings-acl)
Copied to clipboard
Siva Rajesh Kasa, Aniket Goel, Karan Gupta, Sumegh Roychowdhury, Pattisapu Priyatam, Anish Bhanushali, Prasanna Srinivasa Murthy
| Challenge: | Ordinal classification (OC) is a key task in natural language processing with applications in various domains such as sentiment analysis, rating prediction, and more. |
| Approach: | They propose to tackle ordinal classification (OC) through the implicit semantics of the labels . they propose to use a classical explicit approach and an implicit approach that organically engages the semantics. |
| Outcome: | The proposed methods are based on pre-trained language models and offer strategic recommendations based upon specific settings. |
Diverse dialogue generation with context dependent dynamic loss function (2020.coling-main)
Copied to clipboard
| Challenge: | Dialogue systems using deep learning have achieved generation of fluent response sentences to user utterances, but they tend to produce responses that are not diverse and less context-dependent. |
| Approach: | They propose an Inverse N-gram loss function which incorporates contextual fluency and diversity at the same time by a simple formula. |
| Outcome: | The proposed loss function outperforms baseline models in automatic evaluations such as DIST-N and ROUGE and achieves higher scores on human evaluations of coherence and richness. |
ECC: Synergizing Emotion, Cause and Commonsense for Empathetic Dialogue Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Empathy improves human-machine dialogue systems by enhancing the user's experience. |
| Approach: | They propose a framework that leverages specialized encoders to capture the key features of emotion, cause, and commonsense and collaboratively models these through a Conditional Variational Auto-Encoder. |
| Outcome: | Empirical results show that the framework outperforms baseline models and offers a robust solution for empathetic dialogue generation. |
Learning a Matching Model with Co-teaching for Multi-turn Response Selection in Retrieval-based Dialogue Systems (P19-1)
Copied to clipboard
| Challenge: | Existing methods for learning a robust matching model from noisy training data are retrieval-based or generation-based. |
| Approach: | They propose a general co-teaching framework that learns matching models from noisy training data. |
| Outcome: | The proposed learning framework can improve existing models on two public data sets. |
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2020.acl-main)
Copied to clipboard
| Challenge: | Unlikelihood is a technique developed for removal of repetition in language model completions . it allows for a model to be generalized to solve a number of problems . |
| Approach: | They extend the unlikelihood objective to generate generations that contain repetitions . they show that such an objective can be used to improve logical consistency . |
| Outcome: | The proposed approach can be applied to a number of dialogue tasks. |
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA (2020.acl-main)
Copied to clipboard
| Challenge: | Recent years have witnessed a paradigm shift in the way we get our information, and a lot of it. |
| Approach: | They propose a video question answering model which integrates multi-modal input sources and finds temporally relevant information to answer questions. |
| Outcome: | The proposed model outperforms the state-of-the-art on a TVQA dataset. |
Similarity-Based Reconstruction Loss for Meaning Representation (D18-1)
Copied to clipboard
| Challenge: | a new paper proposes and evaluates a set of loss functions that can be used to train models for representation learning . cross-entropy loss penalizes models when they fail to generate the exact word from ground truth data . |
| Approach: | They propose and evaluate loss functions that can be used to train any neural model for representation learning. |
| Outcome: | The proposed loss functions amplify semantic diversity while preserving original meaning . they show performance improvement on paraphrase detection and language inference tasks . |
Jointly Learning to Align and Summarize for Neural Cross-Lingual Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual summarization focus on pipeline methods and training end-to-end models. |
| Approach: | They propose to jointly learn to align and align to train a neural cross-lingual summarization model by using a large-scale corpus. |
| Outcome: | The proposed model outperforms competing models in most cases and can generate cross-lingual summaries without access to any cross-linguistic corpus. |
Time-dependent Entity Embedding is not All You Need: A Re-evaluation of Temporal Knowledge Graph Completion Models under a Unified Framework (2021.emnlp-main)
Copied to clipboard
| Challenge: | Various temporal knowledge graph (KG) completion models have been proposed . knowledge graphs are typically static and store facts in their current state . |
| Approach: | They propose to use temporal embeddings and a score function to model temporal knowledge graphs . they classify the temporal embedded methods into two classes: timestamp and time-dependent . |
| Outcome: | The proposed models outperform current models on ICEWS datasets with 3000 experiments and 13159 GPU hours. |
Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multi-label text classification is a challenging task because it requires capturing label dependencies. |
| Approach: | They propose to use distribution-balanced loss functions to solve label dependency problems in multi-label text classification by capturing label dependencies from a fixed-set of labels. |
| Outcome: | The proposed loss function addresses both the class imbalance and label linkage problems and outperforms other loss functions. |
Rethinking Negative Pairs in Code Search (2023.emnlp-main)
Copied to clipboard
| Challenge: | Comparative learning is a key component in fine-tuning code search models . however, negative samples of InfoNCE may deteriorate its representation learning . |
| Approach: | They propose a loss function that inserts weight terms into InfoNCE to improve contrastive learning. |
| Outcome: | The proposed loss function is a special case of Soft-InfoNCE, the authors show . it is more accurate than other loss functions, and it is faster than other models. |
Not All Terms Matter: Recall-Oriented Adaptive Learning for PLM-aided Query Expansion in Open-Domain Question Answering (2025.acl-long)
Copied to clipboard
| Challenge: | Open-domain question answering (ODQA) systems typically adopt a retriever-reader architecture, where the retriever finds relevant documents, and the reader extracts or synthesizes answers. |
| Approach: | They propose a method that iteratively adjusts the importance weights of QE terms based on their relevance, refining term distinction and enhancing the separation of relevant terms. |
| Outcome: | The proposed method improves retrieval accuracy and overall performance on four ODQA datasets and five QE methods. |
Probing Relative Interaction and Dynamic Calibration in Multi-modal Entity Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Current methods for multi-modal entity alignment ignore relative interactions between modalities and the accuracy of weights. |
| Approach: | They propose a relative interaction and calibration framework for multi-modal entity alignment that uses attention mechanisms to perceive the uncertainty of the weight for each modality. |
| Outcome: | The proposed framework outperforms baselines across 5 datasets and 23 settings. |
Training BERT Models to Carry over a Coding System Developed on One Corpus to Another (2024.lrec-main)
Copied to clipboard
| Challenge: | a pilot project aims to track trends in the perception of literary translation around the 1989 political transformation in Hungary. |
| Approach: | They train BERT models to carry over a coding system developed on a journal to another . aim is to track trends in perception of literary translation around 1989 political transformation . |
| Outcome: | The proposed system can carry over from one coding system to another, the authors show . the system can improve performance and provide better predictions from an ensemble . |
Learning on Imbalanced Noisy Data via Debiased Sample Selection and LLM-Driven Annotation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to learning with noisy labels are prone to selection bias and training bias . obtaining large-scale high-quality datasets is expensive and time-consuming in practical scenarios . |
| Approach: | They propose an imbalanced learning with noisy labels task to let model learn from noisy labels . they first conduct debiased sample selection to better separate clean samples from noisy samples . then they feed selected clean samples to active annotator large language models for re-annotating noisy samples. |
| Outcome: | The proposed method is superior to existing methods on synthetic and real-world datasets. |
Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-modal image-text retrieval struggle with nuanced cross-modal relationships. |
| Approach: | They propose a set-based approach that represents each sample with multiple embeddings to capture nuanced and diverse relationships. |
| Outcome: | The proposed method achieves state-of-the-art performance on MS-COCO and Flickr30k without external data. |
Towards Unified Multimodal Large Language Models: A survey (2026.findings-acl)
Copied to clipboard
| Challenge: | unified multimodal large language models (MLLMs) are emerging but lack a systematic framework to connect them and situate current trends within a broader landscape. |
| Approach: | They present a systematic review of unified Multimodal Large Language Models . they outline the foundational concepts and prerequisites for understanding them . |
| Outcome: | The present review provides a systematic and systematic overview of unified MLLMs . it discusses persistent challenges and identify promising directions for future research . |